1. Field of Invention
The invention relates generally to digital image processing systems. Video-editing workflow methods and apparatus thereof are described.
2. Description of Relevant Art
When a user performs image operations using digital image processing programs such as Adobe Photoshop™ or PhotoDeluxe™, image operations are performed directly on the raw pixels of the image. Since most imaging applications only perform image operations on one resolution, usually the highest resolution, these operations are sometimes very slow, even on the fastest computers. If an application could work at a lower resolution for display purposes, the processing time would significantly decrease, thus increasing the productivity of the user. While it is sometimes possible for an application to work on lower-resolution image data, when the image with all the applied image operations is to be saved, the full-resolution image data must be processed at that time. If this step were not performed, the saved image would only contain low-resolution image data. While this is one option, this is not desired since it would not be possible to obtain an image rasterized at a higher resolution.
In general, an application is forced to generate one resolution (usually the highest), even though lower-resolution data could be used since the output may be targeted to a low-resolution display device and not a high-resolution printer. Unfortunately, it is not possible for the application to know apriori how the image will be used in the future, thus a high-resolution image is usually created.
A more desirable solution would be if a low-resolution (i.e. resultant) image could be generated based on the sequence of image operations, but with “knowledge” about the original high-resolution image (i.e. the digital negative) as well as the list of image operations (i.e. the edit list). The digital negative is the user's original, unmodified image stored at the highest resolution possible. The edit list is a sequence of image operations or image transforms that are to be applied to an image. The resultant image is the result of applying the specified edit list applied to the digital negative at some specified resolution, which includes a reference (i.e. internal or external link) to the digital negative along with the optional edit list.
There have been recent developments in technology that have attempted to solve this problem. An imaging application that supports FlashPix™ technology, developed by a consortium of companies comprising of the Digital Imaging Group (DIG), can display and manipulate on-screen images at a lower-resolution, but retains access to the higher-resolution image data. The FlashPix™ file contains the following items: (a) the original higher-resolution image data, (b) a very limited set of image transforms (such as, rotation, translation, cropping, color twist, blur/sharpen adjustment, brightness/contrast adjustment), and (c) an optional resultant image. While this appears to resolve the problems listed above, there are several limitations. The most important of which is that the original image data, the image transforms, and the optional resultant image are stored together in one central file. This does not provide for a distributed imaging architecture as describe by the invention, nor is it a lightweight solution needed for transmission of images in a low-bandwidth network environment. Additional limitations of FlashPix are that the file format is based on a proprietary, complex format referred to as COM/Structured Storage which, in the current form used by FlashPix™, does not support the ideas put forth by this invention. Further, the image transforms supported are limited to a few operations. The solution described by this invention provides a much more flexible model.
Other attempts have been made that try to remedy the problem outlined above, but with limited success. The Internet Imaging Protocol (IIP) has been developed that provides for optimized access and display of a FlashPix files over the network. When using IIP, a FlashPix file that resides on a serve can be efficiently rendered. A client (i.e. Web browser) must contain executable code (such as a Java applet, ActiveX control, or browser plug-in) that allows it to request portions of the FlashPix file. This involves downloading of fully rendered tiles or portions of the image at different resolutions. While this does provide for efficient display of a FlashPix file over a low bandwidth network connection, it does not solve the generate problem of linking the low-resolution resultant image file back to the high-resolution digital negative and edit list. IIP is simply a method for serving up and viewing large FlashPix files over the network in an efficient manner.
Other variants of FlashPix exist, including LivePicture's FITS technology. The FITS format allows the image operations to be performed on low-resolution image data and saved, thus eliminating the need to do a time consuming render of the high-resolution image data when saving the file. The re-rendering at high resolution is done once the output resolution is determined. However, it does not solve the general problem of linking the low-resolution resultant image file back to the high-resolution digital negative and edit list.
In addition to providing general improvements for distribution of images, there are several web-based imaging solutions that are greatly enhanced by incorporation of this invention, but up until now could not provide similar novel functionality.
A variety of companies offer creation and distribution of on-line greeting cards (Blue Mountain Arts, Yahoo, etc.) and calendars. Some also offer the ability to customize cards and calendars by compositing a personal photo with their content. In general, these are offered as a free service and utilize low-resolution content and/or user's photos. Even if a high-resolution photograph were uploaded, it might be downsampled and composited with the greeting card artwork. This composition, in low-resolution form, is then sent, possibly through e-mail, to the recipient of the card or calendar.
This becomes even more important as higher-quality on-line content providers (i.e. Corbis, Getty, etc.) enter into this space. These companies can offer much higher quality content, but expect to be compensated if this content is taken advantage of. One model might involve users creating greeting cards or calendars, optionally composited with their photos, but only low-resolution resultant output (and optionally with a watermark) to be generated and distributed for free.
Peer-to-peer Web-based file sharing is a model that is becoming increasingly popular. Several applications now exist today, including Gnutella (OpenSource application for peer-to-peer file sharing) and Napster (peer-to-peer audio sharing). A same opportunity exists for direct peer-to-peer sharing of image files. As with any of these applications, and more problematic for images, is that the size of the files could result in an almost unusable user experience when image data is transferred over low-bandwidth network connections. Since pure peer-to-peer (no central intermediate image cache) involves direct connection between two or more end-user's computing devices, the requirement of transferring a large amount of image data, even on a high-bandwidth connection, almost makes this idea infeasible.
For photos today shared using direct peer-to-peer connections, the option is either send the high-resolution image that results in slow transmission or a low-resolution image that is faster, but poorer quality. There is no direct way to send a low resolution image, but with an automatic method of accessing the high-resolution image data if needed.
Embedded imaging devices can also take advantage of this invention to provide increased functionality and performance not possible today. Today certain digital cameras, including the Kodak DC265 and DC290, provide in-camera image processing software support. With this software, a user can apply specific effects or other image processing operations to an image after it has been captured. Although only a small thumbnail of the image can be viewed on the LCD screen, the full resolution image must be processed and stored in the camera memory (such as a CompactFlash card). While it might be possible to perform an optimization step that allows the low-resolution thumbnail to be generated so it can be quickly displayed on the LCD screen, the full-resolution image must still be processed and stored in the camera memory. Even worse, if the original image is also to be retained, two copies of the high-resolution image must be stored (the original and the one with the image operation applied).
In the realm of video image processing. Adobe Premiere and AfterEffects are products that provide powerful real-time video editing. For these and related products, users edit low-resolution (lower frame rate or less image resolution) video sequences to makes cuts, adding transitions and special effects, as well as other video editing operations. The commands are captured and then executed on the full-resolution video stream into a desired video format (such as Quick time, NTSC, or PAL) at a specified target resolution. While this does provide for on-demand rasterization based on the target output device, it is done so in a centralized manner. Inside an Adobe Premiere or AfterEffects project file, all the information and links to the necessary original (high-resolution) video clips are required and must be accessible.
While this approach is standard in the field of video editing, it requires a massive amount of storage space and processing power to render full-resolution, high-quality video output. Even with a high-end system, it still could take a long time (10 minutes to several hours) too fully render the complete video clip. This solution is not viable for a consumer-based video-editing device.
What is desired is an alternative approach for creating a video-editing workflow that requires much less processing power for end-users.